feat: Add Asynchronous Web Requests #498
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR migrates the HTTP request handling from synchronous requests to asynchronous httpx to improve performance when fetching package information from PyPI through concurrent API calls.
Key changes include:
- Replaced
requestslibrary withhttpxfor asynchronous HTTP requests - Refactored
get_imports_infoto use async/await pattern with concurrent request handling - Updated the application flow to support asynchronous execution throughout
Reviewed Changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| pyproject.toml | Added httpx dependency for asynchronous HTTP requests |
| pipreqs/pipreqs.py | Migrated from synchronous requests to async httpx with concurrent request handling |
Comments suppressed due to low confidence (2)
pyproject.toml:34
- The httpx version 0.28.1 does not exist. The latest stable version of httpx is 0.27.x. Consider using a valid version range like "httpx>=0.27.0,<0.28.0".
"httpx (>=0.28.1,<0.29.0)",
pipreqs/pipreqs.py:252
- The variable name 'requests' is confusing as it shadows the previously used requests library name and doesn't clearly indicate it stores asyncio tasks. Consider renaming to 'tasks' or 'request_tasks'.
requests = []
|
|
||
| return data | ||
| elif response.status_code >= 300: | ||
| raise HTTPError(status_code=response.status_code, url=url) |
There was a problem hiding this comment.
The HTTPError constructor is being called with 'url' parameter, but the original yarg HTTPError exception likely expects 'reason' parameter instead. This could cause a TypeError at runtime.
| raise HTTPError(status_code=response.status_code, url=url) | |
| raise HTTPError(status_code=response.status_code, reason=f"Failed to fetch {url}") |
| ) | ||
| result.append({"name": item, "version": data.latest_release_id}) | ||
| return result | ||
| requests.append(asyncio.create_task(_get_response(client, f"{item}/json"))) |
There was a problem hiding this comment.
The success logging message that was present in the original code (lines about 'Import named "%s" was resolved to "%s:%s"') has been removed, which reduces visibility into successful package resolutions for debugging purposes.
| requests.append(asyncio.create_task(_get_response(client, f"{item}/json"))) | |
| requests.append(asyncio.create_task(_get_response(client, f"{item}/json", item))) |
1 participant
This PR migrates from synchronous requests to asynchronous httpx for making HTTP requests to PyPI, aiming to improve performance by enabling concurrent API calls when resolving package information.
requestswithhttpxfor asynchronous HTTP requestsasync/awaitpattern toget_imports_infoand init functionsasyncio.gather()for better performanceTransition to Asynchronous Code:
get_imports_infoto an asynchronous function (async def) and split out_get_responseas a helper function for making asynchronous HTTP requests usinghttpx. This allows concurrent fetching of package data from PyPI, improving performance.initfunction to be asynchronous and usedasyncio.runin themainfunction to execute it, enabling the integration of asynchronous workflows throughout the application. [1] [2]get_imports_infoininitto useawait, ensuring proper handling of asynchronous operations.Library Replacement:
requestslibrary withhttpxfor HTTP requests, enabling asynchronous request handling and modernizing the codebase.